{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Test _check_relevance_with_conversation Method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "import os\n",
    "\n",
    "from autocoder.rag.long_context_rag import LongContextRAG\n",
    "from autocoder.common import AutoCoderArgs\n",
    "import byzerllm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m2024-08-29 15:51:15.965\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mbyzerllm.utils.connect_ray\u001b[0m:\u001b[36mconnect_cluster\u001b[0m:\u001b[36m48\u001b[0m - \u001b[1mJDK 21 will be used (/Users/allwefantasy/.auto-coder/jdk-21.0.2.jdk/Contents/Home)...\u001b[0m\n",
      "2024-08-29 15:51:16,003\tINFO worker.py:1564 -- Connecting to existing Ray cluster at address: 127.0.0.1:6379...\n",
      "2024-08-29 15:51:16,022\tINFO worker.py:1740 -- Connected to Ray cluster. View the dashboard at \u001b[1m\u001b[32m127.0.0.1:8265 \u001b[39m\u001b[22m\n",
      "\u001b[32m2024-08-29 15:51:17.015\u001b[0m | \u001b[31m\u001b[1mERROR   \u001b[0m | \u001b[36mautocoder.rag.long_context_rag\u001b[0m:\u001b[36m_retrieve_documents\u001b[0m:\u001b[36m386\u001b[0m - \u001b[31m\u001b[1mError processing file /Users/allwefantasy/Downloads/docs/.DS_Store: 'utf-8' codec can't decode byte 0x80 in position 3131: invalid start byte\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "# Initialize ByzerLLM and LongContextRAG\n",
    "llm = byzerllm.ByzerLLM.from_default_model(\"deepseek_chat\")\n",
    "args = AutoCoderArgs()\n",
    "rag = LongContextRAG(llm, args, path=\"/Users/allwefantasy/Downloads/docs\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test case 1 result: yes/10\n"
     ]
    }
   ],
   "source": [
    "# Test case 1: Relevant conversation and document\n",
    "conversations1 = [\n",
    "    {\"role\": \"user\", \"content\": \"What are the main features of Python?\"},\n",
    "    {\"role\": \"assistant\", \"content\": \"Python has many features. Can you be more specific about what you're interested in?\"},\n",
    "    {\"role\": \"user\", \"content\": \"I'm interested in Python's support for object-oriented programming.\"}\n",
    "]\n",
    "document1 = \"Python is a high-level, interpreted programming language. It supports multiple programming paradigms, including structured, object-oriented, and functional programming. Python's support for object-oriented programming includes features like classes, inheritance, and polymorphism.\"\n",
    "\n",
    "result1 = rag._check_relevance_with_conversation.with_llm(llm).run(conversations1, [document1])\n",
    "print(\"Test case 1 result:\", result1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test case 2 result: no/0\n"
     ]
    }
   ],
   "source": [
    "# Test case 2: Irrelevant conversation and document\n",
    "conversations2 = [\n",
    "    {\"role\": \"user\", \"content\": \"What's the weather like today?\"},\n",
    "    {\"role\": \"assistant\", \"content\": \"I'm sorry, but I don't have real-time weather information. Can you please check a weather website or app for the most up-to-date forecast?\"},\n",
    "    {\"role\": \"user\", \"content\": \"Okay, thanks. How about the forecast for tomorrow?\"}\n",
    "]\n",
    "document2 = \"The Pythagorean theorem states that in a right-angled triangle, the square of the length of the hypotenuse is equal to the sum of squares of the other two sides.\"\n",
    "\n",
    "result2 = rag._check_relevance_with_conversation.with_llm(llm).run(conversations2, [document2])\n",
    "print(\"Test case 2 result:\", result2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test case 3 result: yes/7\n"
     ]
    },
    {
     "ename": "",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31mThe Kernel crashed while executing code in the current cell or a previous cell. \n",
      "\u001b[1;31mPlease review the code in the cell(s) to identify a possible cause of the failure. \n",
      "\u001b[1;31mClick <a href='https://aka.ms/vscodeJupyterKernelCrash'>here</a> for more info. \n",
      "\u001b[1;31mView Jupyter <a href='command:jupyter.viewOutput'>log</a> for further details."
     ]
    }
   ],
   "source": [
    "# Test case 3: Partially relevant conversation and document\n",
    "conversations3 = [\n",
    "    {\"role\": \"user\", \"content\": \"Can you explain the concept of machine learning?\"},\n",
    "    {\"role\": \"assistant\", \"content\": \"Certainly! Machine learning is a subset of artificial intelligence that focuses on building systems that can learn from and make decisions based on data. What specific aspect of machine learning would you like to know more about?\"},\n",
    "    {\"role\": \"user\", \"content\": \"I'd like to know about supervised learning algorithms.\"}\n",
    "]\n",
    "document3 = \"Machine learning is a field of artificial intelligence that uses statistical techniques to give computer systems the ability to 'learn' from data, without being explicitly programmed. There are three main types of machine learning: supervised learning, unsupervised learning, and reinforcement learning.\"\n",
    "\n",
    "result3 = rag._check_relevance_with_conversation.with_llm(llm).run(conversations3, [document3])\n",
    "print(\"Test case 3 result:\", result3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In these test cases, we've created three scenarios:\n",
    "\n",
    "1. A relevant conversation and document about Python's object-oriented programming features.\n",
    "2. An irrelevant conversation about weather and a document about the Pythagorean theorem.\n",
    "3. A partially relevant conversation about machine learning and a document that covers machine learning broadly but doesn't specifically focus on supervised learning.\n",
    "\n",
    "The results should show whether the `_check_relevance_with_conversation` method correctly identifies the relevance of the documents to the conversations. You may need to adjust the interpretation of the results based on the exact format of the method's output."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
